Use Yutori native tool sets - #15
Merged
Merged
Conversation
rgarcia
marked this pull request as ready for review
May 14, 2026 15:20
|
Firetiger deploy monitoring skipped This PR didn't match the auto-monitor filter configured on your GitHub connection:
Reason: PR modifies AI/CUA tool sets and Yutori provider integration, not kernel API endpoints or Temporal workflows as specified in the filter. To monitor this PR anyway, reply with |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Inconsistent modifier key naming within same file
- Updated
clear_before_typingkeypress generation to use the same normalizedctrlmodifier mapping used byhold_keyspaths and adjusted the unit test expectation accordingly.
- Updated
Or push these changes by commenting:
@cursor push 3049347a8e
Preview (3049347a8e)
diff --git a/packages/ai/src/providers/yutori/actions.ts b/packages/ai/src/providers/yutori/actions.ts
--- a/packages/ai/src/providers/yutori/actions.ts
+++ b/packages/ai/src/providers/yutori/actions.ts
@@ -195,7 +195,7 @@
if (text === undefined) return undefined;
const actions: CuaAction[] = [];
if (args.clear_before_typing === true) {
- actions.push({ type: "keypress", keys: ["Control", "a"] }, { type: "keypress", keys: ["Backspace"] });
+ actions.push({ type: "keypress", keys: [MODIFIER_MAP.control, "a"] }, { type: "keypress", keys: ["Backspace"] });
}
actions.push({ type: "type", text });
if (args.press_enter_after === true) actions.push({ type: "keypress", keys: ["Enter"] });
diff --git a/packages/ai/test/yutori-actions.test.ts b/packages/ai/test/yutori-actions.test.ts
--- a/packages/ai/test/yutori-actions.test.ts
+++ b/packages/ai/test/yutori-actions.test.ts
@@ -26,7 +26,7 @@
{ type: "drag", path: [{ x: 100, y: 200 }, { x: 300, y: 400 }], button: "left" },
]);
expect(yutori.toCanonicalActions("type", { text: "hello", clear_before_typing: true, press_enter_after: true })).toEqual([
- { type: "keypress", keys: ["Control", "a"] },
+ { type: "keypress", keys: ["ctrl", "a"] },
{ type: "keypress", keys: ["Backspace"] },
{ type: "type", text: "hello" },
{ type: "keypress", keys: ["Enter"] },You can send follow-ups to the cloud agent here.
rgarcia
commented
May 14, 2026
| const SCROLL_AMOUNT_PER_NOTCH = 120; | ||
| const DEFAULT_WAIT_MS = 2000; | ||
| const NAVIGATION_WAIT_MS = 1500; | ||
| const GOTO_WAIT_MS = 2000; |
Contributor
Author
There was a problem hiding this comment.
comment on what these consts control and rhyme/reason
| const NAVIGATION_WAIT_MS = 1500; | ||
| const GOTO_WAIT_MS = 2000; | ||
|
|
||
| const MODIFIER_MAP: Record<string, string> = { |
Contributor
Author
There was a problem hiding this comment.
this seems like somethign we should pull out to a common layer--presumably other providers will spit out modifier keys of all kinds and we want to normalize to a canonical set, similar to what we do for actions
different from actions though i'd be fine doing this normalization in some logic that is basically a catchall / union of everything we may come across across all providers
remember that the canonical modifier set should be x11 keysyms since that's what kernel uses under the hood
| super: "super", | ||
| }; | ||
|
|
||
| export function createComputerToolDefinitions(_options?: unknown): [] { |
Contributor
Author
There was a problem hiding this comment.
comment explaining why this is empty
| ); | ||
| } | ||
|
|
||
| export function toCanonicalActions(name: string, args: Record<string, unknown>): CuaAction[] | undefined { |
Contributor
Author
There was a problem hiding this comment.
should have a test that this is exhaustive, at least for YUTORI_N15_CORE_ACTION_TYPES
and we should be flexible to expand the CuaAction common types for things not modeled
| export const COMPUTER_TOOL_COORDINATES = { type: "normalized", range: [0, 1000] } as const satisfies ComputerToolCoordinateSystem; | ||
|
|
||
| export const YUTORI_INSTRUCTIONS_RAW = `You control a Kernel cloud browser. Prefer batched computer actions for browser interaction and include screenshot or URL reads when you need updated state.`; | ||
| export const YUTORI_INSTRUCTIONS_RAW = ""; |
Contributor
Author
There was a problem hiding this comment.
comment on why (yutori recommends this, link to docs)
Contributor
Author
|
addressed the review comments in
verified:
|
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Duplicate key alias maps already diverge on entries
- The translator now reuses
normalizeCuaKeyfrom@onkernel/cua-aiandsuper_lwas added toCUA_KEY_ALIASES, eliminating the divergent alias mapping path.
- The translator now reuses
- ✅ Fixed: Exported functions and types lack TSDoc comments
- Added TSDoc comments for the flagged exported types and normalization/mapping helpers in
common.tsandyutori/actions.ts.
- Added TSDoc comments for the flagged exported types and normalization/mapping helpers in
Or push these changes by commenting:
@cursor push 02e28e8ce1
Preview (02e28e8ce1)
diff --git a/packages/agent/src/translator/translator.ts b/packages/agent/src/translator/translator.ts
--- a/packages/agent/src/translator/translator.ts
+++ b/packages/agent/src/translator/translator.ts
@@ -1,5 +1,6 @@
import type Kernel from "@onkernel/sdk";
import type { BrowserCreateResponse, BrowserRetrieveResponse } from "@onkernel/sdk/resources/browsers";
+import { normalizeCuaKey } from "@onkernel/cua-ai";
import type { BatchExecutionResult, ModelAction } from "./types";
export type KernelBrowser = BrowserCreateResponse | BrowserRetrieveResponse;
@@ -230,48 +231,13 @@
};
}
-const KEY_ALIASES: Record<string, string> = {
- ctrl: "Control_L",
- control: "Control_L",
- control_l: "Control_L",
- controlleft: "Control_L",
- alt: "Alt_L",
- alt_l: "Alt_L",
- altleft: "Alt_L",
- shift: "Shift_L",
- shift_l: "Shift_L",
- shiftleft: "Shift_L",
- meta: "Super_L",
- super: "Super_L",
- super_l: "Super_L",
- cmd: "Super_L",
- command: "Super_L",
- enter: "Return",
- return: "Return",
- escape: "Escape",
- esc: "Escape",
- backspace: "BackSpace",
- delete: "Delete",
- tab: "Tab",
- space: "space",
- left: "Left",
- right: "Right",
- up: "Up",
- down: "Down",
-};
-
function translateKeys(keys: string[]): string[] {
return keys.flatMap((key) =>
key
.split("+")
.map((part) => part.trim())
.filter(Boolean)
- .map((part) => {
- const alias = KEY_ALIASES[part.replace(/[-\s]/g, "_").toLowerCase()];
- if (alias) return alias;
- if (part.length === 1 && part >= "A" && part <= "Z") return part.toLowerCase();
- return part;
- }),
+ .map((part) => normalizeCuaKey(part)),
);
}
diff --git a/packages/ai/src/providers/common.ts b/packages/ai/src/providers/common.ts
--- a/packages/ai/src/providers/common.ts
+++ b/packages/ai/src/providers/common.ts
@@ -135,6 +135,7 @@
export const CUA_MODIFIER_KEYSYMS = ["Control_L", "Alt_L", "Shift_L", "Super_L"] as const;
+/** Canonical X11 keysyms accepted by `hold_keys` and modifier-only keypresses. */
export type CuaModifierKeysym = (typeof CUA_MODIFIER_KEYSYMS)[number];
const CUA_KEY_ALIASES: Record<string, string> = {
@@ -169,6 +170,7 @@
shiftleft: "Shift_L",
space: "space",
super: "Super_L",
+ super_l: "Super_L",
tab: "Tab",
up: "Up",
};
@@ -191,11 +193,13 @@
return trimmed;
}
+/** Resolve a raw key string to a canonical modifier keysym when possible. */
export function normalizeCuaModifierKey(value: string): CuaModifierKeysym | undefined {
const key = normalizeCuaKey(value);
return CUA_MODIFIER_KEYSYM_SET.has(key) ? (key as CuaModifierKeysym) : undefined;
}
+/** Normalize a `+`-delimited key combination (for example `ctrl+shift+tab`). */
export function normalizeCuaKeyCombo(value: string): string[] {
return value
.split("+")
@@ -203,6 +207,7 @@
.filter(Boolean);
}
+/** Normalize whitespace-delimited key combinations into sequential chord presses. */
export function normalizeCuaKeySequence(value: string): string[][] {
return value
.split(/\s+/)
diff --git a/packages/ai/src/providers/yutori/actions.ts b/packages/ai/src/providers/yutori/actions.ts
--- a/packages/ai/src/providers/yutori/actions.ts
+++ b/packages/ai/src/providers/yutori/actions.ts
@@ -85,9 +85,13 @@
...YUTORI_N15_EXPANDED_ACTION_TYPES,
] as const;
+/** Native action names emitted by Yutori Navigator n1. */
export type YutoriN1ActionType = (typeof YUTORI_N1_ACTION_TYPES)[number];
+/** Native action names emitted by Yutori Navigator n1.5 core tool set. */
export type YutoriN15CoreActionType = (typeof YUTORI_N15_CORE_ACTION_TYPES)[number];
+/** Native action names emitted by Yutori Navigator n1.5 expanded tool set. */
export type YutoriN15ExpandedActionType = (typeof YUTORI_N15_EXPANDED_ACTION_TYPES)[number];
+/** Union of all provider-native action names this adapter can normalize. */
export type YutoriNativeActionType = YutoriN1ActionType | YutoriN15CoreActionType | YutoriN15ExpandedActionType;
const YUTORI_NATIVE_ACTION_NAMES = new Set<string>([
@@ -110,18 +114,22 @@
return [];
}
+/** Return the Yutori `tool_set` id required by a model, if any. */
export function yutoriToolSetForModel(modelId: string): typeof YUTORI_N15_CORE_TOOL_SET | undefined {
return modelId.startsWith("n1.5") ? YUTORI_N15_CORE_TOOL_SET : undefined;
}
+/** Return the provider-native action names available for the selected model. */
export function yutoriNativeActionsForModel(modelId: string): readonly YutoriNativeActionType[] {
return modelId.startsWith("n1.5") ? YUTORI_N15_CORE_ACTION_TYPES : YUTORI_N1_ACTION_TYPES;
}
+/** Check whether a tool call name belongs to Yutori's native action vocabulary. */
export function isYutoriNativeActionName(name: string): boolean {
return YUTORI_NATIVE_ACTION_NAMES.has(name);
}
+/** Check whether a tool name is one of CUA's default computer-use tools. */
export function isCuaDefaultToolName(name: string): boolean {
return (
name === CUA_BATCH_TOOL_NAME ||
@@ -130,6 +138,7 @@
);
}
+/** Convert a Yutori native action payload into canonical CUA actions. */
export function toCanonicalActions(name: string, args: Record<string, unknown>): CuaAction[] | undefined {
const coords = readPoint(args.coordinates);
switch (name) {
@@ -184,10 +193,12 @@
}
}
+/** Resolve canonical tool call name from a canonical CUA action. */
export function canonicalToolCallName(action: CuaAction): CuaActionType {
return action.type;
}
+/** Strip the action discriminant and return canonical tool arguments. */
export function canonicalToolCallArguments(action: CuaAction): Record<string, unknown> {
const { type: _type, ...args } = action as CuaAction & Record<string, unknown>;
return args;
diff --git a/packages/ai/test/common-keys.test.ts b/packages/ai/test/common-keys.test.ts
--- a/packages/ai/test/common-keys.test.ts
+++ b/packages/ai/test/common-keys.test.ts
@@ -10,6 +10,7 @@
it("normalizes common provider key names to X11 keysyms", () => {
expect(normalizeCuaModifierKey("ctrl")).toBe("Control_L");
expect(normalizeCuaModifierKey("command")).toBe("Super_L");
+ expect(normalizeCuaModifierKey("super_l")).toBe("Super_L");
expect(normalizeCuaKey("Backspace")).toBe("BackSpace");
expect(normalizeCuaKey("ArrowLeft")).toBe("Left");
expect(normalizeCuaKey("enter")).toBe("Return");You can send follow-ups to the cloud agent here.
| export function canonicalToolCallArguments(action: CuaAction): Record<string, unknown> { | ||
| const { type: _type, ...args } = action as CuaAction & Record<string, unknown>; | ||
| return args; | ||
| } |
There was a problem hiding this comment.
Exported functions and types lack TSDoc comments
Low Severity
Multiple new exported functions (yutoriToolSetForModel, yutoriNativeActionsForModel, isYutoriNativeActionName, isCuaDefaultToolName, toCanonicalActions, canonicalToolCallName, canonicalToolCallArguments) and exported types (YutoriN1ActionType, YutoriN15CoreActionType, YutoriNativeActionType, CuaModifierKeysym) lack TSDoc comments. Exported functions like normalizeCuaModifierKey, normalizeCuaKeyCombo, and normalizeCuaKeySequence in common.ts are also missing TSDoc. This violates the rule requiring TSDoc on all exported types, interfaces, and classes.
Additional Locations (1)
Triggered by learned rule: Exported types, interfaces, and classes require TSDoc
Reviewed by Cursor Bugbot for commit 807c63c. Configure here.
Contributor
Author
|
Updated this PR with the follow-up Yutori runtime work:
Validation:
|
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
There are 3 total unresolved issues (including 1 from previous review).
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Exported function
isYutoriNativeActionNameis never used- Removed the unused
isYutoriNativeActionNameexport and its backing set from Yutori actions to eliminate dead code.
- Removed the unused
- ✅ Fixed: Duplicate
normalizeGotoUrlacross two packages- Extracted a single shared
normalizeGotoUrlhelper inproviders/commonand updated both Yutori actions and agent translator to use it.
- Extracted a single shared
Or push these changes by commenting:
@cursor push 09af3d5ea8
Preview (09af3d5ea8)
diff --git a/packages/agent/src/translator/translator.ts b/packages/agent/src/translator/translator.ts
--- a/packages/agent/src/translator/translator.ts
+++ b/packages/agent/src/translator/translator.ts
@@ -1,6 +1,6 @@
import type Kernel from "@onkernel/sdk";
import type { BrowserCreateResponse, BrowserRetrieveResponse } from "@onkernel/sdk/resources/browsers";
-import type { ComputerToolCoordinateSystem, CuaScreenshotSpec } from "@onkernel/cua-ai";
+import { normalizeGotoUrl, type ComputerToolCoordinateSystem, type CuaScreenshotSpec } from "@onkernel/cua-ai";
import sharp from "sharp";
import { isKernelModifierKey, normalizeKernelKey, normalizeKernelKeyCombo } from "./keys";
import type { BatchExecutionResult, ModelAction } from "./types";
@@ -98,7 +98,7 @@
continue;
}
if (type === "goto") {
- const url = normalizeGotoUrl(stringOr(action.url, ""));
+ const url = normalizeGotoUrl(stringOr(action.url, "")) ?? "";
pending.push(
keypress(["Control", "l"]),
{ type: "type_text", type_text: { text: url } },
@@ -241,12 +241,6 @@
return typeof value === "string" && value.length > 0 ? value : fallback;
}
-function normalizeGotoUrl(value: string): string {
- const url = value.trim();
- if (!url || /^[a-z][a-z0-9+.-]*:\/\//i.test(url)) return url;
- return `https://${url}`;
-}
-
function clickMouseButtonOr(value: unknown, fallback: ClickMouseButton): ClickMouseButton {
const candidate = stringOr(value, fallback);
if (candidate === "left" || candidate === "right" || candidate === "middle" || candidate === "back" || candidate === "forward") {
diff --git a/packages/ai/src/providers/common.ts b/packages/ai/src/providers/common.ts
--- a/packages/ai/src/providers/common.ts
+++ b/packages/ai/src/providers/common.ts
@@ -293,6 +293,13 @@
export const CUA_NAVIGATION_TOOL_DESCRIPTION = "High-level browser navigation helpers for goto, back, forward, and url.";
+export function normalizeGotoUrl(value: unknown): string | undefined {
+ if (typeof value !== "string") return undefined;
+ const url = value.trim();
+ if (!url) return undefined;
+ return /^[a-z][a-z0-9+.-]*:\/\//i.test(url) ? url : `https://${url}`;
+}
+
export interface CreateComputerToolDefinitionsOptions {
actions?: readonly CuaActionType[];
}
diff --git a/packages/ai/src/providers/yutori/actions.ts b/packages/ai/src/providers/yutori/actions.ts
--- a/packages/ai/src/providers/yutori/actions.ts
+++ b/packages/ai/src/providers/yutori/actions.ts
@@ -3,6 +3,7 @@
CUA_BATCH_TOOL_NAME,
CUA_NAVIGATION_TOOL_NAME,
createCuaActionToolDefinitions,
+ normalizeGotoUrl,
type CuaAction,
type CuaActionType,
} from "../common";
@@ -104,11 +105,6 @@
export type YutoriN15ExpandedActionType = (typeof YUTORI_N15_EXPANDED_ACTION_TYPES)[number];
export type YutoriNativeActionType = YutoriN1ActionType | YutoriN15CoreActionType | YutoriN15ExpandedActionType;
-const YUTORI_NATIVE_ACTION_NAMES = new Set<string>([
- ...YUTORI_N1_ACTION_TYPES,
- ...YUTORI_N15_ACTION_TYPES,
-]);
-
const DEFAULT_SCROLL_AMOUNT = 3;
const SCROLL_AMOUNT_PER_NOTCH = 120;
const DEFAULT_WAIT_MS = 2000;
@@ -132,10 +128,6 @@
return modelId.startsWith("n1.5") ? YUTORI_N15_CORE_ACTION_TYPES : YUTORI_N1_ACTION_TYPES;
}
-export function isYutoriNativeActionName(name: string): boolean {
- return YUTORI_NATIVE_ACTION_NAMES.has(name);
-}
-
export function isCuaDefaultToolName(name: string): boolean {
return (
name === CUA_BATCH_TOOL_NAME ||
@@ -270,13 +262,6 @@
return key ? { hold_keys: [key] } : {};
}
-function normalizeGotoUrl(value: unknown): string | undefined {
- if (typeof value !== "string") return undefined;
- const url = value.trim();
- if (!url) return undefined;
- return /^[a-z][a-z0-9+.-]*:\/\//i.test(url) ? url : `https://${url}`;
-}
-
function secondsToMs(value: unknown, fallback: number): number {
if (typeof value !== "number" || !Number.isFinite(value) || value <= 0) return fallback;
return Math.round(value * 1000);
diff --git a/packages/ai/src/providers/yutori/index.ts b/packages/ai/src/providers/yutori/index.ts
--- a/packages/ai/src/providers/yutori/index.ts
+++ b/packages/ai/src/providers/yutori/index.ts
@@ -2,7 +2,6 @@
export {
createComputerToolDefinitions,
- isYutoriNativeActionName,
toCanonicalActions,
yutoriNativeActionsForModel,
yutoriToolSetForModel,You can send follow-ups to the cloud agent here.
rgarcia
force-pushed
the
hypeship/yutori-native-tools
branch
from
May 15, 2026 17:04
0565d6f to
c6ed21c
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
There are 3 total unresolved issues (including 1 from previous review).
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Yutori tool-set payload hook runs twice per request
- Added a per-payload symbol marker so
yutoriNativeToolSetOnPayloadapplies only once across internal and composed hook paths, and strip the marker before sending the request.
- Added a per-payload symbol marker so
- ✅ Fixed: New exported interface lacks TSDoc documentation
- Added interface-level and field-level TSDoc comments to
CuaScreenshotTransformSpecdescribing its screenshot transform contract.
- Added interface-level and field-level TSDoc comments to
Or push these changes by commenting:
@cursor push c4ea5804c4
Preview (c4ea5804c4)
diff --git a/packages/ai/src/providers/yutori/provider.ts b/packages/ai/src/providers/yutori/provider.ts
--- a/packages/ai/src/providers/yutori/provider.ts
+++ b/packages/ai/src/providers/yutori/provider.ts
@@ -44,8 +44,13 @@
return yutoriNativeToolSetOnPayload(payload, model);
}
+const YUTORI_NATIVE_TOOL_SET_APPLIED = Symbol("yutoriNativeToolSetApplied");
+
+type YutoriPayload = Record<string, unknown> & { [YUTORI_NATIVE_TOOL_SET_APPLIED]?: true };
+
export function yutoriNativeToolSetOnPayload(payload: unknown, model?: Model<Api>): unknown | undefined {
if (!payload || typeof payload !== "object") return undefined;
+ if ((payload as YutoriPayload)[YUTORI_NATIVE_TOOL_SET_APPLIED]) return undefined;
const current = payload as { tools?: unknown };
const tools = Array.isArray(current.tools)
? current.tools.filter((tool) => {
@@ -58,7 +63,8 @@
...(payload as Record<string, unknown>),
...(toolSet ? { tool_set: toolSet, disable_tools: [...YUTORI_N15_EXPANDED_ACTION_TYPES] } : {}),
...(tools && tools.length > 0 ? { tools } : { tools: undefined }),
- };
+ [YUTORI_NATIVE_TOOL_SET_APPLIED]: true,
+ } satisfies YutoriPayload;
}
async function runYutoriStream(
@@ -84,9 +90,14 @@
};
const tools = convertTools(context);
if (tools.length > 0) payload.tools = tools;
- payload = yutoriNativeToolSetOnPayload(payload, model) as Record<string, unknown>;
+ const nativePayload = yutoriNativeToolSetOnPayload(payload, model);
+ if (nativePayload !== undefined) payload = nativePayload as Record<string, unknown>;
const nextPayload = await options?.onPayload?.(payload, model);
if (nextPayload !== undefined) payload = nextPayload as Record<string, unknown>;
+ const payloadWithMarker = payload as YutoriPayload;
+ if (payloadWithMarker[YUTORI_NATIVE_TOOL_SET_APPLIED]) {
+ delete payloadWithMarker[YUTORI_NATIVE_TOOL_SET_APPLIED];
+ }
const { data: response, response: rawResponse } = await client.chat.completions
.create(payload as unknown as Parameters<typeof client.chat.completions.create>[0], { signal: options?.signal })
diff --git a/packages/ai/src/runtime-spec.ts b/packages/ai/src/runtime-spec.ts
--- a/packages/ai/src/runtime-spec.ts
+++ b/packages/ai/src/runtime-spec.ts
@@ -7,10 +7,17 @@
import * as yutori from "./providers/yutori/index";
import type { ComputerToolCoordinateSystem } from "./providers/common";
+/**
+ * Image transform options applied to captured screenshots before provider upload.
+ */
export interface CuaScreenshotTransformSpec {
+ /** Target image width in pixels. */
width: number;
+ /** Target image height in pixels. */
height: number;
+ /** Output encoding sent to the provider. */
format: "png" | "jpeg" | "webp";
+ /** Optional lossy quality setting used by `jpeg` and `webp` encoders. */
quality?: number;
}
diff --git a/packages/ai/test/yutori-payload.test.ts b/packages/ai/test/yutori-payload.test.ts
--- a/packages/ai/test/yutori-payload.test.ts
+++ b/packages/ai/test/yutori-payload.test.ts
@@ -30,6 +30,15 @@
expect(next.tools).toBeUndefined();
});
+ it("does not re-apply the native tool-set transform to an already transformed payload", () => {
+ const payload = {
+ tools: [{ type: "function", function: { name: "batch_computer_actions" } }],
+ };
+ const next = yutori.yutoriNativeToolSetOnPayload(payload, { id: "n1.5-latest" } as never);
+ const reapplied = yutori.yutoriNativeToolSetOnPayload(next, { id: "n1.5-latest" } as never);
+ expect(reapplied).toBeUndefined();
+ });
+
it("returns undefined for non-object payloads", () => {
expect(yutori.yutoriBuiltinToolsOnPayload(undefined)).toBeUndefined();
expect(yutori.yutoriBuiltinToolsOnPayload("x")).toBeUndefined();You can send follow-ups to the cloud agent here.
rgarcia
force-pushed
the
hypeship/yutori-native-tools
branch
5 times, most recently
from
May 16, 2026 19:24
4046268 to
64e9f90
Compare
rgarcia
force-pushed
the
hypeship/yutori-native-tools
branch
from
May 16, 2026 19:31
64e9f90 to
0c46fdb
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 4 total unresolved issues (including 3 from previous reviews).
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Multi-action expansion creates mismatched tool call IDs
- Normalized expanded Yutori tool-call IDs back to their original IDs in outbound assistant/tool messages and collapsed duplicate expanded tool results to a single upstream response.
Or push these changes by commenting:
@cursor push 4bcaf1dd5f
Preview (4bcaf1dd5f)
diff --git a/packages/ai/src/providers/yutori/provider.ts b/packages/ai/src/providers/yutori/provider.ts
--- a/packages/ai/src/providers/yutori/provider.ts
+++ b/packages/ai/src/providers/yutori/provider.ts
@@ -184,6 +184,7 @@
function convertMessages(context: Context): ChatCompletionMessageParam[] {
const messages: ChatCompletionMessageParam[] = [];
+ const expandedToolCallIdMap = new Map<string, string>();
if (context.systemPrompt) messages.push({ role: "system", content: context.systemPrompt });
for (const message of context.messages) {
if (message.role === "user") {
@@ -196,29 +197,58 @@
.filter((part): part is TextContent => part.type === "text")
.map((part) => part.text)
.join("");
+ const seenExpandedToolCallIds = new Set<string>();
const toolCalls = message.content
.filter((part): part is ToolCall => part.type === "toolCall")
- .map((part) => ({
- id: part.id,
- type: "function" as const,
- function: { name: part.name, arguments: JSON.stringify(part.arguments ?? {}) },
- }));
+ .flatMap((part) => {
+ const originalId = originalYutoriToolCallId(part.id, part.name);
+ if (originalId !== part.id) {
+ expandedToolCallIdMap.set(part.id, originalId);
+ if (seenExpandedToolCallIds.has(originalId)) return [];
+ seenExpandedToolCallIds.add(originalId);
+ }
+ return [
+ {
+ id: originalId,
+ type: "function" as const,
+ function: { name: part.name, arguments: JSON.stringify(part.arguments ?? {}) },
+ },
+ ];
+ });
messages.push({
role: "assistant",
content: text || null,
...(toolCalls.length > 0 ? { tool_calls: toolCalls } : {}),
});
} else if (message.role === "toolResult") {
- messages.push({
+ const toolCallId = expandedToolCallIdMap.get(message.toolCallId) ?? message.toolCallId;
+ const toolMessage = {
role: "tool",
- tool_call_id: message.toolCallId,
+ tool_call_id: toolCallId,
content: message.content.map(toOpenAIContentPart) as unknown as string,
- });
+ } as ChatCompletionMessageParam;
+ const previous = messages[messages.length - 1];
+ if (toolCallId !== message.toolCallId && isToolMessageParam(previous) && previous.tool_call_id === toolCallId) {
+ messages[messages.length - 1] = toolMessage;
+ } else {
+ messages.push(toolMessage);
+ }
}
}
return messages;
}
+function originalYutoriToolCallId(id: string, name: string): string {
+ if (!isYutoriLocalActionToolName(name)) return id;
+ const match = id.match(/^(.*)_\d+$/);
+ if (!match?.[1]) return id;
+ return match[1];
+}
+
+function isToolMessageParam(message: ChatCompletionMessageParam | undefined): message is ChatCompletionMessageParam & { role: "tool" } {
+ return Boolean(message && message.role === "tool");
+}
+
function convertTools(context: Context): Array<Record<string, unknown>> {
return (context.tools ?? []).map((tool) => ({
type: "function",You can send follow-ups to the cloud agent here.
rgarcia
force-pushed
the
hypeship/yutori-native-tools
branch
2 times, most recently
from
May 19, 2026 16:07
51f3aea to
93fa003
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
There are 5 total unresolved issues (including 3 from previous reviews).
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Tzafon readNumber makes coordinate validation dead code
- Changed Tzafon
readNumberto returnundefinedfor missing/non-finite values and added explicit usage defaults so coordinate checks and fallback fields now work as intended.
- Changed Tzafon
- ✅ Fixed: Duplicate canonicalToolCallName/Arguments across Tzafon and Yutori providers
- Removed the duplicated Tzafon helper implementations and imported the existing exported canonical helper functions from
yutori/actions.ts.
- Removed the duplicated Tzafon helper implementations and imported the existing exported canonical helper functions from
Or push these changes by commenting:
@cursor push 88b05b5349
Preview (88b05b5349)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -45,7 +45,7 @@
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
TZAFON_API_KEY: ${{ secrets.TZAFON_API_KEY }}
YUTORI_API_KEY: ${{ secrets.YUTORI_API_KEY }}
- run: npm test --workspace @onkernel/cua-ai -- test/batch-tool.integration.test.ts
+ run: npm test --workspace @onkernel/cua-ai -- test/computer-tool.integration.test.ts
agent-e2e:
runs-on: ubuntu-latest
diff --git a/package-lock.json b/package-lock.json
--- a/package-lock.json
+++ b/package-lock.json
@@ -736,6 +736,16 @@
"zod-to-json-schema": "^3.25.0"
}
},
+ "node_modules/@emnapi/runtime": {
+ "version": "1.10.0",
+ "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.10.0.tgz",
+ "integrity": "sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==",
+ "license": "MIT",
+ "optional": true,
+ "dependencies": {
+ "tslib": "^2.4.0"
+ }
+ },
"node_modules/@esbuild/aix-ppc64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.27.7.tgz",
@@ -1201,6 +1211,471 @@
}
}
},
+ "node_modules/@img/colour": {
+ "version": "1.1.0",
+ "resolved": "https://registry.npmjs.org/@img/colour/-/colour-1.1.0.tgz",
+ "integrity": "sha512-Td76q7j57o/tLVdgS746cYARfSyxk8iEfRxewL9h4OMzYhbW4TAcppl0mT4eyqXddh6L/jwoM75mo7ixa/pCeQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@img/sharp-darwin-arm64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-darwin-arm64/-/sharp-darwin-arm64-0.34.5.tgz",
+ "integrity": "sha512-imtQ3WMJXbMY4fxb/Ndp6HBTNVtWCUI0WdobyheGf5+ad6xX8VIDO8u2xE4qc/fr08CKG/7dDseFtn6M6g/r3w==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-darwin-arm64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-darwin-x64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-darwin-x64/-/sharp-darwin-x64-0.34.5.tgz",
+ "integrity": "sha512-YNEFAF/4KQ/PeW0N+r+aVVsoIY0/qxxikF2SWdp+NRkmMB7y9LBZAVqQ4yhGCm/H3H270OSykqmQMKLBhBJDEw==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-darwin-x64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-libvips-darwin-arm64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-arm64/-/sharp-libvips-darwin-arm64-1.2.4.tgz",
+ "integrity": "sha512-zqjjo7RatFfFoP0MkQ51jfuFZBnVE2pRiaydKJ1G/rHZvnsrHAOcQALIi9sA5co5xenQdTugCvtb1cuf78Vf4g==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-darwin-x64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-x64/-/sharp-libvips-darwin-x64-1.2.4.tgz",
+ "integrity": "sha512-1IOd5xfVhlGwX+zXv2N93k0yMONvUlANylbJw1eTah8K/Jtpi15KC+WSiaX/nBmbm2HxRM1gZ0nSdjSsrZbGKg==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-arm": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm/-/sharp-libvips-linux-arm-1.2.4.tgz",
+ "integrity": "sha512-bFI7xcKFELdiNCVov8e44Ia4u2byA+l3XtsAj+Q8tfCwO6BQ8iDojYdvoPMqsKDkuoOo+X6HZA0s0q11ANMQ8A==",
+ "cpu": [
+ "arm"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-arm64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm64/-/sharp-libvips-linux-arm64-1.2.4.tgz",
+ "integrity": "sha512-excjX8DfsIcJ10x1Kzr4RcWe1edC9PquDRRPx3YVCvQv+U5p7Yin2s32ftzikXojb1PIFc/9Mt28/y+iRklkrw==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-ppc64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-ppc64/-/sharp-libvips-linux-ppc64-1.2.4.tgz",
+ "integrity": "sha512-FMuvGijLDYG6lW+b/UvyilUWu5Ayu+3r2d1S8notiGCIyYU/76eig1UfMmkZ7vwgOrzKzlQbFSuQfgm7GYUPpA==",
+ "cpu": [
+ "ppc64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-riscv64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-riscv64/-/sharp-libvips-linux-riscv64-1.2.4.tgz",
+ "integrity": "sha512-oVDbcR4zUC0ce82teubSm+x6ETixtKZBh/qbREIOcI3cULzDyb18Sr/Wcyx7NRQeQzOiHTNbZFF1UwPS2scyGA==",
+ "cpu": [
+ "riscv64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-s390x": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-s390x/-/sharp-libvips-linux-s390x-1.2.4.tgz",
+ "integrity": "sha512-qmp9VrzgPgMoGZyPvrQHqk02uyjA0/QrTO26Tqk6l4ZV0MPWIW6LTkqOIov+J1yEu7MbFQaDpwdwJKhbJvuRxQ==",
+ "cpu": [
+ "s390x"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-x64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-x64/-/sharp-libvips-linux-x64-1.2.4.tgz",
+ "integrity": "sha512-tJxiiLsmHc9Ax1bz3oaOYBURTXGIRDODBqhveVHonrHJ9/+k89qbLl0bcJns+e4t4rvaNBxaEZsFtSfAdquPrw==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linuxmusl-arm64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-arm64/-/sharp-libvips-linuxmusl-arm64-1.2.4.tgz",
+ "integrity": "sha512-FVQHuwx1IIuNow9QAbYUzJ+En8KcVm9Lk5+uGUQJHaZmMECZmOlix9HnH7n1TRkXMS0pGxIJokIVB9SuqZGGXw==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linuxmusl-x64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-x64/-/sharp-libvips-linuxmusl-x64-1.2.4.tgz",
+ "integrity": "sha512-+LpyBk7L44ZIXwz/VYfglaX/okxezESc6UxDSoyo2Ks6Jxc4Y7sGjpgU9s4PMgqgjj1gZCylTieNamqA1MF7Dg==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-linux-arm": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm/-/sharp-linux-arm-0.34.5.tgz",
+ "integrity": "sha512-9dLqsvwtg1uuXBGZKsxem9595+ujv0sJ6Vi8wcTANSFpwV/GONat5eCkzQo/1O6zRIkh0m/8+5BjrRr7jDUSZw==",
+ "cpu": [
+ "arm"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-arm": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linux-arm64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm64/-/sharp-linux-arm64-0.34.5.tgz",
+ "integrity": "sha512-bKQzaJRY/bkPOXyKx5EVup7qkaojECG6NLYswgktOZjaXecSAeCWiZwwiFf3/Y+O1HrauiE3FVsGxFg8c24rZg==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-arm64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linux-ppc64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-ppc64/-/sharp-linux-ppc64-0.34.5.tgz",
+ "integrity": "sha512-7zznwNaqW6YtsfrGGDA6BRkISKAAE1Jo0QdpNYXNMHu2+0dTrPflTLNkpc8l7MUP5M16ZJcUvysVWWrMefZquA==",
+ "cpu": [
+ "ppc64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-ppc64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linux-riscv64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-riscv64/-/sharp-linux-riscv64-0.34.5.tgz",
+ "integrity": "sha512-51gJuLPTKa7piYPaVs8GmByo7/U7/7TZOq+cnXJIHZKavIRHAP77e3N2HEl3dgiqdD/w0yUfiJnII77PuDDFdw==",
+ "cpu": [
+ "riscv64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-riscv64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linux-s390x": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-s390x/-/sharp-linux-s390x-0.34.5.tgz",
+ "integrity": "sha512-nQtCk0PdKfho3eC5MrbQoigJ2gd1CgddUMkabUj+rBevs8tZ2cULOx46E7oyX+04WGfABgIwmMC0VqieTiR4jg==",
+ "cpu": [
+ "s390x"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-s390x": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linux-x64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-x64/-/sharp-linux-x64-0.34.5.tgz",
+ "integrity": "sha512-MEzd8HPKxVxVenwAa+JRPwEC7QFjoPWuS5NZnBt6B3pu7EG2Ge0id1oLHZpPJdn3OQK+BQDiw9zStiHBTJQQQQ==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-x64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linuxmusl-arm64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-arm64/-/sharp-linuxmusl-arm64-0.34.5.tgz",
+ "integrity": "sha512-fprJR6GtRsMt6Kyfq44IsChVZeGN97gTD331weR1ex1c1rypDEABN6Tm2xa1wE6lYb5DdEnk03NZPqA7Id21yg==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linuxmusl-arm64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linuxmusl-x64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-x64/-/sharp-linuxmusl-x64-0.34.5.tgz",
+ "integrity": "sha512-Jg8wNT1MUzIvhBFxViqrEhWDGzqymo3sV7z7ZsaWbZNDLXRJZoRGrjulp60YYtV4wfY8VIKcWidjojlLcWrd8Q==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linuxmusl-x64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-wasm32": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-wasm32/-/sharp-wasm32-0.34.5.tgz",
+ "integrity": "sha512-OdWTEiVkY2PHwqkbBI8frFxQQFekHaSSkUIJkwzclWZe64O1X4UlUjqqqLaPbUpMOQk6FBu/HtlGXNblIs0huw==",
+ "cpu": [
+ "wasm32"
+ ],
+ "license": "Apache-2.0 AND LGPL-3.0-or-later AND MIT",
+ "optional": true,
+ "dependencies": {
+ "@emnapi/runtime": "^1.7.0"
+ },
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-win32-arm64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-win32-arm64/-/sharp-win32-arm64-0.34.5.tgz",
+ "integrity": "sha512-WQ3AgWCWYSb2yt+IG8mnC6Jdk9Whs7O0gxphblsLvdhSpSTtmu69ZG1Gkb6NuvxsNACwiPV6cNSZNzt0KPsw7g==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "Apache-2.0 AND LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-win32-ia32": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-win32-ia32/-/sharp-win32-ia32-0.34.5.tgz",
+ "integrity": "sha512-FV9m/7NmeCmSHDD5j4+4pNI8Cp3aW+JvLoXcTUo0IqyjSfAZJ8dIUmijx1qaJsIiU+Hosw6xM5KijAWRJCSgNg==",
+ "cpu": [
+ "ia32"
+ ],
+ "license": "Apache-2.0 AND LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-win32-x64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-win32-x64/-/sharp-win32-x64-0.34.5.tgz",
+ "integrity": "sha512-+29YMsqY2/9eFEiW93eqWnuLcWcufowXewwSNIT6UwZdUUCrM3oFjMWH/Z6/TMmb4hlFenmfAVbpWeup2jryCw==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "Apache-2.0 AND LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
"node_modules/@jridgewell/sourcemap-codec": {
"version": "1.5.5",
"resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.5.tgz",
@@ -2848,6 +3323,15 @@
"node": ">= 14"
}
},
+ "node_modules/detect-libc": {
+ "version": "2.1.2",
+ "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",
+ "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",
+ "license": "Apache-2.0",
+ "engines": {
+ "node": ">=8"
+ }
+ },
"node_modules/diff": {
"version": "8.0.4",
"license": "BSD-3-Clause",
@@ -3957,6 +4441,62 @@
],
"license": "MIT"
},
+ "node_modules/semver": {
+ "version": "7.8.0",
+ "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.0.tgz",
+ "integrity": "sha512-AcM7dV/5ul4EekoQ29Agm5vri8JNqRyj39o0qpX6vDF2GZrtutZl5RwgD1XnZjiTAfncsJhMI48QQH3sN87YNA==",
+ "license": "ISC",
+ "bin": {
+ "semver": "bin/semver.js"
+ },
+ "engines": {
+ "node": ">=10"
+ }
+ },
+ "node_modules/sharp": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/sharp/-/sharp-0.34.5.tgz",
+ "integrity": "sha512-Ou9I5Ft9WNcCbXrU9cMgPBcCK8LiwLqcbywW3t4oDV37n1pzpuNLsYiAV8eODnjbtQlSDwZ2cUEeQz4E54Hltg==",
+ "hasInstallScript": true,
+ "license": "Apache-2.0",
+ "dependencies": {
+ "@img/colour": "^1.0.0",
+ "detect-libc": "^2.1.2",
+ "semver": "^7.7.3"
+ },
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-darwin-arm64": "0.34.5",
+ "@img/sharp-darwin-x64": "0.34.5",
+ "@img/sharp-libvips-darwin-arm64": "1.2.4",
+ "@img/sharp-libvips-darwin-x64": "1.2.4",
+ "@img/sharp-libvips-linux-arm": "1.2.4",
+ "@img/sharp-libvips-linux-arm64": "1.2.4",
+ "@img/sharp-libvips-linux-ppc64": "1.2.4",
+ "@img/sharp-libvips-linux-riscv64": "1.2.4",
+ "@img/sharp-libvips-linux-s390x": "1.2.4",
+ "@img/sharp-libvips-linux-x64": "1.2.4",
+ "@img/sharp-libvips-linuxmusl-arm64": "1.2.4",
+ "@img/sharp-libvips-linuxmusl-x64": "1.2.4",
+ "@img/sharp-linux-arm": "0.34.5",
+ "@img/sharp-linux-arm64": "0.34.5",
+ "@img/sharp-linux-ppc64": "0.34.5",
+ "@img/sharp-linux-riscv64": "0.34.5",
+ "@img/sharp-linux-s390x": "0.34.5",
+ "@img/sharp-linux-x64": "0.34.5",
+ "@img/sharp-linuxmusl-arm64": "0.34.5",
+ "@img/sharp-linuxmusl-x64": "0.34.5",
+ "@img/sharp-wasm32": "0.34.5",
+ "@img/sharp-win32-arm64": "0.34.5",
+ "@img/sharp-win32-ia32": "0.34.5",
+ "@img/sharp-win32-x64": "0.34.5"
+ }
+ },
"node_modules/siginfo": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/siginfo/-/siginfo-2.0.0.tgz",
@@ -4623,6 +5163,7 @@
"@earendil-works/pi-ai": "^0.74.0",
"@onkernel/cua-ai": "0.1.0",
"@onkernel/sdk": "0.49.0",
+ "sharp": "^0.34.5",
"typebox": "^1.1.38"
},
"devDependencies": {
diff --git a/packages/agent/README.md b/packages/agent/README.md
--- a/packages/agent/README.md
+++ b/packages/agent/README.md
@@ -84,6 +84,9 @@
- `browser` (Kernel browser response)
- `client` (Kernel SDK client)
- CUA model refs (`"provider:model"`) accepted where pi expects a concrete model
+- `extraTools` for caller-owned tools appended after built-in CUA tools
+- `synthesizeBatchTool: true` to add a local `batch_computer_actions` helper
+- `computerUseExtraTool: true` to add the local `computer_use_extra` helper
If auth callbacks are omitted, both classes default to CUA env var conventions:
- OpenAI: `OPENAI_API_KEY`
@@ -94,10 +97,15 @@
### Tool Defaults
-If tools are omitted, the classes install canonical CUA computer tool executors
-using runtime specs from `@onkernel/cua-ai`. If tools are provided, they are
-used exactly.
+By default, the classes install provider-selected canonical CUA computer tool
+executors using runtime specs from `@onkernel/cua-ai`. Use `extraTools` to add
+caller-owned tools alongside the provider's computer-use tools.
+`batch_computer_actions` and `computer_use_extra` are opt-in CuaAgent and
+CuaAgentHarness sugar. They are synthesized from the provider's canonical
+action definitions and can be enabled with `synthesizeBatchTool` and
+`computerUseExtraTool`.
+
### Model Switching
`CuaAgent` follows pi `Agent` semantics: assign `agent.state.model` to a
@@ -124,6 +132,7 @@
browser,
client,
toolDefinitions: runtime.toolDefinitions,
+ synthesizeBatchTool: true,
}),
myCustomTool,
];
diff --git a/packages/agent/package.json b/packages/agent/package.json
--- a/packages/agent/package.json
+++ b/packages/agent/package.json
@@ -44,6 +44,7 @@
"@earendil-works/pi-ai": "^0.74.0",
"@onkernel/cua-ai": "0.1.0",
"@onkernel/sdk": "0.49.0",
+ "sharp": "^0.34.5",
"typebox": "^1.1.38"
},
"devDependencies": {
diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -11,6 +11,8 @@
} from "./vendor/pi-agent-core/index";
import {
type Api,
+ CUA_BATCH_TOOL_NAME,
+ CUA_NAVIGATION_TOOL_NAME,
type CuaModelRef,
getCuaEnvApiKey,
type Model,
@@ -20,7 +22,7 @@
} from "@onkernel/cua-ai";
import type Kernel from "@onkernel/sdk";
import { createCuaComputerTools } from "./tools";
-import type { KernelBrowser } from "./translator/translator";
+import { InternalComputerTranslator, type KernelBrowser } from "./translator/translator";
/** A CUA model reference string or a concrete pi model object. */
type CuaRuntimeInput = CuaModelRef | Model<Api>;
@@ -45,8 +47,6 @@
type CuaAgentInitialState = Omit<NonNullable<AgentOptions["initialState"]>, "model" | "tools"> & {
/** Model to use for the first turn. CUA refs are resolved before pi sees the state. */
model: CuaRuntimeInput;
- /** Optional caller-owned tools. Omit this to install the provider's default CUA tools. */
- tools?: AgentTool[];
};
/**
@@ -63,6 +63,12 @@
client: Kernel;
/** Initial pi state plus a CUA-aware model value. */
initialState: CuaAgentInitialState;
+ /** Additional caller-owned tools appended after built-in CUA tools. */
+ extraTools?: AgentTool[];
+ /** Add a local batch_computer_actions helper synthesized from provider action definitions. */
+ synthesizeBatchTool?: boolean;
+ /** Add the local computer_use_extra navigation helper. */
+ computerUseExtraTool?: boolean;
};
/**
@@ -75,13 +81,19 @@
export type CuaAgentHarnessOptions<
TSkill extends Skill = Skill,
TPromptTemplate extends PromptTemplate = PromptTemplate,
-> = Omit<AgentHarnessOptions<TSkill, TPromptTemplate, AgentTool>, "model"> & {
+> = Omit<AgentHarnessOptions<TSkill, TPromptTemplate, AgentTool>, "model" | "tools"> & {
/** Kernel browser session used by default CUA tools. */
browser: KernelBrowser;
/** Kernel SDK client used by default CUA tools. */
client: Kernel;
/** Model used by the harness. CUA refs are resolved before pi sees the model. */
model: CuaRuntimeInput;
+ /** Additional caller-owned tools appended after built-in CUA tools. */
+ extraTools?: AgentTool[];
+ /** Add a local batch_computer_actions helper synthesized from provider action definitions. */
+ synthesizeBatchTool?: boolean;
+ /** Add the local computer_use_extra navigation helper. */
+ computerUseExtraTool?: boolean;
/** Optional payload hook composed after the provider-specific CUA payload hook. */
onPayload?: SimpleStreamOptions["onPayload"];
};
@@ -89,9 +101,10 @@
/**
* Holds the CUA-specific pieces that have to change when a model changes.
*
- * If callers omit `tools` or `systemPrompt`, CUA owns those values and refreshes
- * them from `@onkernel/cua-ai` whenever the model changes. If callers pass
- * their own tools or prompt, the controller preserves those caller-owned values.
+ * CUA owns the computer-use tools and refreshes them from `@onkernel/cua-ai`
+ * whenever the model changes. Caller-owned `extraTools` are appended after
+ * those defaults. If callers pass their own prompt, the controller preserves
+ * that caller-owned prompt.
*/
class CuaRuntimeController {
private runtimeSpec: CuaRuntimeSpec;
@@ -101,7 +114,9 @@
browser: KernelBrowser;
client: Kernel;
model: CuaRuntimeInput;
- tools?: AgentTool[];
+ extraTools?: AgentTool[];
+ synthesizeBatchTool?: boolean;
+ computerUseExtraTool?: boolean;
systemPrompt?: unknown;
onPayload?: SimpleStreamOptions["onPayload"];
},
@@ -114,7 +129,7 @@
}
get ownsTools(): boolean {
- return this.options.tools === undefined;
+ return true;
}
get ownsSystemPrompt(): boolean {
@@ -130,20 +145,80 @@
}
tools(): AgentTool[] {
- return (
- this.options.tools ??
- createCuaComputerTools({
+ return [
+ ...createCuaComputerTools({
browser: this.options.browser,
client: this.options.client,
toolDefinitions: this.runtimeSpec.toolDefinitions,
- })
- );
+ coordinateSystem: this.runtimeSpec.coordinateSystem,
+ screenshot: this.runtimeSpec.screenshot,
+ synthesizeBatchTool: this.options.synthesizeBatchTool,
+ computerUseExtraTool: this.options.computerUseExtraTool,
+ }),
+ ...(this.options.extraTools ?? []),
+ ];
}
onPayloadFor(model: CuaRuntimeInput): SimpleStreamOptions["onPayload"] {
const runtimeSpec = resolveCuaRuntimeSpec(model);
- return composeOnPayload(runtimeSpec.onPayload, this.options.onPayload);
+ return composeOnPayload(
+ composeOnPayload(this.screenshotOnPayload(runtimeSpec), this.providerOnPayload(runtimeSpec)),
+ this.options.onPayload,
+ );
}
+
+ keepToolNames(): string[] {
+ return [
+ ...(this.options.extraTools ?? []).map((tool) => tool.name),
+ ...(this.options.synthesizeBatchTool ? [CUA_BATCH_TOOL_NAME] : []),
+ ...(this.options.computerUseExtraTool ? [CUA_NAVIGATION_TOOL_NAME] : []),
+ ];
+ }
+
+ private providerOnPayload(runtimeSpec: CuaRuntimeSpec): SimpleStreamOptions["onPayload"] | undefined {
+ if (!runtimeSpec.onPayload) return undefined;
+ return async (payload, model) =>
+ runtimeSpec.onPayload?.(payload, model as Model<Api>, { keepToolNames: this.keepToolNames() });
+ }
+
+ private screenshotOnPayload(runtimeSpec: CuaRuntimeSpec): SimpleStreamOptions["onPayload"] | undefined {
+ if (!runtimeSpec.screenshot?.appendToLatestMessage) return undefined;
+ return async (payload) => {
+ if (!payload || typeof payload !== "object") return undefined;
+ const current = payload as { messages?: unknown };
+ if (!Array.isArray(current.messages) || current.messages.length === 0) return undefined;
+ const last = current.messages[current.messages.length - 1];
+ if (!last || typeof last !== "object") return undefined;
+ const lastMessage = last as { content?: unknown; role?: unknown };
+ if (lastMessage.role !== "user" && lastMessage.role !== "tool") return undefined;
... diff truncated: showing 800 of 3812 linesYou can send follow-ups to the cloud agent here.
rgarcia
force-pushed
the
hypeship/yutori-native-tools
branch
5 times, most recently
from
May 20, 2026 15:19
ab0c7c1 to
360dba8
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Tzafon reuses tool call IDs
- Tzafon now suffixes expanded computer-call IDs when one native action maps to multiple canonical tool calls, preserving unique toolCallId correlation.
Or push these changes by commenting:
@cursor push 0520d9fe7d
Preview (0520d9fe7d)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -45,7 +45,7 @@
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
TZAFON_API_KEY: ${{ secrets.TZAFON_API_KEY }}
YUTORI_API_KEY: ${{ secrets.YUTORI_API_KEY }}
- run: npm test --workspace @onkernel/cua-ai -- test/batch-tool.integration.test.ts
+ run: npm test --workspace @onkernel/cua-ai -- test/computer-tool.integration.test.ts
agent-e2e:
runs-on: ubuntu-latest
diff --git a/package-lock.json b/package-lock.json
--- a/package-lock.json
+++ b/package-lock.json
@@ -736,6 +736,16 @@
"zod-to-json-schema": "^3.25.0"
}
},
+ "node_modules/@emnapi/runtime": {
+ "version": "1.10.0",
+ "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.10.0.tgz",
+ "integrity": "sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==",
+ "license": "MIT",
+ "optional": true,
+ "dependencies": {
+ "tslib": "^2.4.0"
+ }
+ },
"node_modules/@esbuild/aix-ppc64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.27.7.tgz",
@@ -1201,6 +1211,471 @@
}
}
},
+ "node_modules/@img/colour": {
+ "version": "1.1.0",
+ "resolved": "https://registry.npmjs.org/@img/colour/-/colour-1.1.0.tgz",
+ "integrity": "sha512-Td76q7j57o/tLVdgS746cYARfSyxk8iEfRxewL9h4OMzYhbW4TAcppl0mT4eyqXddh6L/jwoM75mo7ixa/pCeQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@img/sharp-darwin-arm64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-darwin-arm64/-/sharp-darwin-arm64-0.34.5.tgz",
+ "integrity": "sha512-imtQ3WMJXbMY4fxb/Ndp6HBTNVtWCUI0WdobyheGf5+ad6xX8VIDO8u2xE4qc/fr08CKG/7dDseFtn6M6g/r3w==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-darwin-arm64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-darwin-x64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-darwin-x64/-/sharp-darwin-x64-0.34.5.tgz",
+ "integrity": "sha512-YNEFAF/4KQ/PeW0N+r+aVVsoIY0/qxxikF2SWdp+NRkmMB7y9LBZAVqQ4yhGCm/H3H270OSykqmQMKLBhBJDEw==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-darwin-x64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-libvips-darwin-arm64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-arm64/-/sharp-libvips-darwin-arm64-1.2.4.tgz",
+ "integrity": "sha512-zqjjo7RatFfFoP0MkQ51jfuFZBnVE2pRiaydKJ1G/rHZvnsrHAOcQALIi9sA5co5xenQdTugCvtb1cuf78Vf4g==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-darwin-x64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-x64/-/sharp-libvips-darwin-x64-1.2.4.tgz",
+ "integrity": "sha512-1IOd5xfVhlGwX+zXv2N93k0yMONvUlANylbJw1eTah8K/Jtpi15KC+WSiaX/nBmbm2HxRM1gZ0nSdjSsrZbGKg==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-arm": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm/-/sharp-libvips-linux-arm-1.2.4.tgz",
+ "integrity": "sha512-bFI7xcKFELdiNCVov8e44Ia4u2byA+l3XtsAj+Q8tfCwO6BQ8iDojYdvoPMqsKDkuoOo+X6HZA0s0q11ANMQ8A==",
+ "cpu": [
+ "arm"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-arm64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm64/-/sharp-libvips-linux-arm64-1.2.4.tgz",
+ "integrity": "sha512-excjX8DfsIcJ10x1Kzr4RcWe1edC9PquDRRPx3YVCvQv+U5p7Yin2s32ftzikXojb1PIFc/9Mt28/y+iRklkrw==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-ppc64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-ppc64/-/sharp-libvips-linux-ppc64-1.2.4.tgz",
+ "integrity": "sha512-FMuvGijLDYG6lW+b/UvyilUWu5Ayu+3r2d1S8notiGCIyYU/76eig1UfMmkZ7vwgOrzKzlQbFSuQfgm7GYUPpA==",
+ "cpu": [
+ "ppc64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-riscv64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-riscv64/-/sharp-libvips-linux-riscv64-1.2.4.tgz",
+ "integrity": "sha512-oVDbcR4zUC0ce82teubSm+x6ETixtKZBh/qbREIOcI3cULzDyb18Sr/Wcyx7NRQeQzOiHTNbZFF1UwPS2scyGA==",
+ "cpu": [
+ "riscv64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-s390x": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-s390x/-/sharp-libvips-linux-s390x-1.2.4.tgz",
+ "integrity": "sha512-qmp9VrzgPgMoGZyPvrQHqk02uyjA0/QrTO26Tqk6l4ZV0MPWIW6LTkqOIov+J1yEu7MbFQaDpwdwJKhbJvuRxQ==",
+ "cpu": [
+ "s390x"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-x64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-x64/-/sharp-libvips-linux-x64-1.2.4.tgz",
+ "integrity": "sha512-tJxiiLsmHc9Ax1bz3oaOYBURTXGIRDODBqhveVHonrHJ9/+k89qbLl0bcJns+e4t4rvaNBxaEZsFtSfAdquPrw==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linuxmusl-arm64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-arm64/-/sharp-libvips-linuxmusl-arm64-1.2.4.tgz",
+ "integrity": "sha512-FVQHuwx1IIuNow9QAbYUzJ+En8KcVm9Lk5+uGUQJHaZmMECZmOlix9HnH7n1TRkXMS0pGxIJokIVB9SuqZGGXw==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linuxmusl-x64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-x64/-/sharp-libvips-linuxmusl-x64-1.2.4.tgz",
+ "integrity": "sha512-+LpyBk7L44ZIXwz/VYfglaX/okxezESc6UxDSoyo2Ks6Jxc4Y7sGjpgU9s4PMgqgjj1gZCylTieNamqA1MF7Dg==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-linux-arm": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm/-/sharp-linux-arm-0.34.5.tgz",
+ "integrity": "sha512-9dLqsvwtg1uuXBGZKsxem9595+ujv0sJ6Vi8wcTANSFpwV/GONat5eCkzQo/1O6zRIkh0m/8+5BjrRr7jDUSZw==",
+ "cpu": [
+ "arm"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-arm": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linux-arm64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm64/-/sharp-linux-arm64-0.34.5.tgz",
+ "integrity": "sha512-bKQzaJRY/bkPOXyKx5EVup7qkaojECG6NLYswgktOZjaXecSAeCWiZwwiFf3/Y+O1HrauiE3FVsGxFg8c24rZg==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-arm64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linux-ppc64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-ppc64/-/sharp-linux-ppc64-0.34.5.tgz",
+ "integrity": "sha512-7zznwNaqW6YtsfrGGDA6BRkISKAAE1Jo0QdpNYXNMHu2+0dTrPflTLNkpc8l7MUP5M16ZJcUvysVWWrMefZquA==",
+ "cpu": [
+ "ppc64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-ppc64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linux-riscv64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-riscv64/-/sharp-linux-riscv64-0.34.5.tgz",
+ "integrity": "sha512-51gJuLPTKa7piYPaVs8GmByo7/U7/7TZOq+cnXJIHZKavIRHAP77e3N2HEl3dgiqdD/w0yUfiJnII77PuDDFdw==",
+ "cpu": [
+ "riscv64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-riscv64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linux-s390x": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-s390x/-/sharp-linux-s390x-0.34.5.tgz",
+ "integrity": "sha512-nQtCk0PdKfho3eC5MrbQoigJ2gd1CgddUMkabUj+rBevs8tZ2cULOx46E7oyX+04WGfABgIwmMC0VqieTiR4jg==",
+ "cpu": [
+ "s390x"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-s390x": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linux-x64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-x64/-/sharp-linux-x64-0.34.5.tgz",
+ "integrity": "sha512-MEzd8HPKxVxVenwAa+JRPwEC7QFjoPWuS5NZnBt6B3pu7EG2Ge0id1oLHZpPJdn3OQK+BQDiw9zStiHBTJQQQQ==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-x64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linuxmusl-arm64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-arm64/-/sharp-linuxmusl-arm64-0.34.5.tgz",
+ "integrity": "sha512-fprJR6GtRsMt6Kyfq44IsChVZeGN97gTD331weR1ex1c1rypDEABN6Tm2xa1wE6lYb5DdEnk03NZPqA7Id21yg==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linuxmusl-arm64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linuxmusl-x64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-x64/-/sharp-linuxmusl-x64-0.34.5.tgz",
+ "integrity": "sha512-Jg8wNT1MUzIvhBFxViqrEhWDGzqymo3sV7z7ZsaWbZNDLXRJZoRGrjulp60YYtV4wfY8VIKcWidjojlLcWrd8Q==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linuxmusl-x64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-wasm32": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-wasm32/-/sharp-wasm32-0.34.5.tgz",
+ "integrity": "sha512-OdWTEiVkY2PHwqkbBI8frFxQQFekHaSSkUIJkwzclWZe64O1X4UlUjqqqLaPbUpMOQk6FBu/HtlGXNblIs0huw==",
+ "cpu": [
+ "wasm32"
+ ],
+ "license": "Apache-2.0 AND LGPL-3.0-or-later AND MIT",
+ "optional": true,
+ "dependencies": {
+ "@emnapi/runtime": "^1.7.0"
+ },
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-win32-arm64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-win32-arm64/-/sharp-win32-arm64-0.34.5.tgz",
+ "integrity": "sha512-WQ3AgWCWYSb2yt+IG8mnC6Jdk9Whs7O0gxphblsLvdhSpSTtmu69ZG1Gkb6NuvxsNACwiPV6cNSZNzt0KPsw7g==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "Apache-2.0 AND LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-win32-ia32": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-win32-ia32/-/sharp-win32-ia32-0.34.5.tgz",
+ "integrity": "sha512-FV9m/7NmeCmSHDD5j4+4pNI8Cp3aW+JvLoXcTUo0IqyjSfAZJ8dIUmijx1qaJsIiU+Hosw6xM5KijAWRJCSgNg==",
+ "cpu": [
+ "ia32"
+ ],
+ "license": "Apache-2.0 AND LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-win32-x64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-win32-x64/-/sharp-win32-x64-0.34.5.tgz",
+ "integrity": "sha512-+29YMsqY2/9eFEiW93eqWnuLcWcufowXewwSNIT6UwZdUUCrM3oFjMWH/Z6/TMmb4hlFenmfAVbpWeup2jryCw==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "Apache-2.0 AND LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
"node_modules/@jridgewell/sourcemap-codec": {
"version": "1.5.5",
"resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.5.tgz",
@@ -2848,6 +3323,15 @@
"node": ">= 14"
}
},
+ "node_modules/detect-libc": {
+ "version": "2.1.2",
+ "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",
+ "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",
+ "license": "Apache-2.0",
+ "engines": {
+ "node": ">=8"
+ }
+ },
"node_modules/diff": {
"version": "8.0.4",
"license": "BSD-3-Clause",
@@ -3957,6 +4441,62 @@
],
"license": "MIT"
},
+ "node_modules/semver": {
+ "version": "7.8.0",
+ "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.0.tgz",
+ "integrity": "sha512-AcM7dV/5ul4EekoQ29Agm5vri8JNqRyj39o0qpX6vDF2GZrtutZl5RwgD1XnZjiTAfncsJhMI48QQH3sN87YNA==",
+ "license": "ISC",
+ "bin": {
+ "semver": "bin/semver.js"
+ },
+ "engines": {
+ "node": ">=10"
+ }
+ },
+ "node_modules/sharp": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/sharp/-/sharp-0.34.5.tgz",
+ "integrity": "sha512-Ou9I5Ft9WNcCbXrU9cMgPBcCK8LiwLqcbywW3t4oDV37n1pzpuNLsYiAV8eODnjbtQlSDwZ2cUEeQz4E54Hltg==",
+ "hasInstallScript": true,
+ "license": "Apache-2.0",
+ "dependencies": {
+ "@img/colour": "^1.0.0",
+ "detect-libc": "^2.1.2",
+ "semver": "^7.7.3"
+ },
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-darwin-arm64": "0.34.5",
+ "@img/sharp-darwin-x64": "0.34.5",
+ "@img/sharp-libvips-darwin-arm64": "1.2.4",
+ "@img/sharp-libvips-darwin-x64": "1.2.4",
+ "@img/sharp-libvips-linux-arm": "1.2.4",
+ "@img/sharp-libvips-linux-arm64": "1.2.4",
+ "@img/sharp-libvips-linux-ppc64": "1.2.4",
+ "@img/sharp-libvips-linux-riscv64": "1.2.4",
+ "@img/sharp-libvips-linux-s390x": "1.2.4",
+ "@img/sharp-libvips-linux-x64": "1.2.4",
+ "@img/sharp-libvips-linuxmusl-arm64": "1.2.4",
+ "@img/sharp-libvips-linuxmusl-x64": "1.2.4",
+ "@img/sharp-linux-arm": "0.34.5",
+ "@img/sharp-linux-arm64": "0.34.5",
+ "@img/sharp-linux-ppc64": "0.34.5",
+ "@img/sharp-linux-riscv64": "0.34.5",
+ "@img/sharp-linux-s390x": "0.34.5",
+ "@img/sharp-linux-x64": "0.34.5",
+ "@img/sharp-linuxmusl-arm64": "0.34.5",
+ "@img/sharp-linuxmusl-x64": "0.34.5",
+ "@img/sharp-wasm32": "0.34.5",
+ "@img/sharp-win32-arm64": "0.34.5",
+ "@img/sharp-win32-ia32": "0.34.5",
+ "@img/sharp-win32-x64": "0.34.5"
+ }
+ },
"node_modules/siginfo": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/siginfo/-/siginfo-2.0.0.tgz",
@@ -4623,6 +5163,7 @@
"@earendil-works/pi-ai": "^0.74.0",
"@onkernel/cua-ai": "0.1.0",
"@onkernel/sdk": "0.49.0",
+ "sharp": "^0.34.5",
"typebox": "^1.1.38"
},
"devDependencies": {
diff --git a/packages/agent/README.md b/packages/agent/README.md
--- a/packages/agent/README.md
+++ b/packages/agent/README.md
@@ -84,6 +84,9 @@
- `browser` (Kernel browser response)
- `client` (Kernel SDK client)
- CUA model refs (`"provider:model"`) accepted where pi expects a concrete model
+- `extraTools` to add your own pi tools alongside the built-in browser tools
+- `batchTool: true` to let the model run multiple browser actions in one tool call
+- `computerUseExtra: true` to let the model use a small navigation helper
If auth callbacks are omitted, both classes default to CUA env var conventions:
- OpenAI: `OPENAI_API_KEY`
@@ -94,10 +97,27 @@
### Tool Defaults
-If tools are omitted, the classes install canonical CUA computer tool executors
-using runtime specs from `@onkernel/cua-ai`. If tools are provided, they are
-used exactly.
+By default, the classes install provider-selected canonical CUA computer tool
+executors using runtime specs from `@onkernel/cua-ai`. Use `extraTools` to add
+your own pi tools alongside the provider's computer-use tools. This is useful
+when the model needs to call application-specific code, such as looking up a
+record, writing a database row, or handing off to another service while it also
+controls the browser.
+`batchTool: true` adds the `batch_computer_actions` tool. Use it when you want
+the model to group several browser actions into one call, for example moving,
+clicking, typing, waiting, and then reading a screenshot. The batch tool is
+synthesized from the selected provider's normal browser action definitions, so
+it only batches actions that provider runtime already supports.
+
+`computerUseExtra: true` adds the `computer_use_extra` tool. Use it when you
+want one compact helper for common browser navigation/read operations:
+`goto`, `back`, `forward`, and `url`.
+
+The TypeScript API follows pi's camelCase option style (`extraTools`,
+`batchTool`, `computerUseExtra`). Names like `batch_computer_actions` and
+`computer_use_extra` are the literal tool names the model may see in traces.
+
### Model Switching
`CuaAgent` follows pi `Agent` semantics: assign `agent.state.model` to a
@@ -124,6 +144,7 @@
browser,
client,
toolDefinitions: runtime.toolDefinitions,
+ batchTool: true,
}),
myCustomTool,
];
diff --git a/packages/agent/package.json b/packages/agent/package.json
--- a/packages/agent/package.json
+++ b/packages/agent/package.json
@@ -44,6 +44,7 @@
"@earendil-works/pi-ai": "^0.74.0",
"@onkernel/cua-ai": "0.1.0",
"@onkernel/sdk": "0.49.0",
+ "sharp": "^0.34.5",
"typebox": "^1.1.38"
},
"devDependencies": {
diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -11,6 +11,8 @@
} from "./vendor/pi-agent-core/index";
import {
type Api,
+ CUA_BATCH_TOOL_NAME,
+ CUA_NAVIGATION_TOOL_NAME,
type CuaModelRef,
getCuaEnvApiKey,
type Model,
@@ -20,7 +22,7 @@
} from "@onkernel/cua-ai";
import type Kernel from "@onkernel/sdk";
import { createCuaComputerTools } from "./tools";
-import type { KernelBrowser } from "./translator/translator";
+import { InternalComputerTranslator, type KernelBrowser } from "./translator/translator";
/** A CUA model reference string or a concrete pi model object. */
type CuaRuntimeInput = CuaModelRef | Model<Api>;
@@ -45,8 +47,6 @@
type CuaAgentInitialState = Omit<NonNullable<AgentOptions["initialState"]>, "model" | "tools"> & {
/** Model to use for the first turn. CUA refs are resolved before pi sees the state. */
model: CuaRuntimeInput;
- /** Optional caller-owned tools. Omit this to install the provider's default CUA tools. */
- tools?: AgentTool[];
};
/**
@@ -63,6 +63,12 @@
client: Kernel;
/** Initial pi state plus a CUA-aware model value. */
initialState: CuaAgentInitialState;
+ /** Add your own pi tools alongside the built-in browser tools. */
+ extraTools?: AgentTool[];
+ /** Expose a batch tool so the model can run multiple browser actions in one call. */
+ batchTool?: boolean;
+ /** Expose a helper for browser navigation and URL reads. */
+ computerUseExtra?: boolean;
};
/**
@@ -75,13 +81,19 @@
export type CuaAgentHarnessOptions<
TSkill extends Skill = Skill,
TPromptTemplate extends PromptTemplate = PromptTemplate,
-> = Omit<AgentHarnessOptions<TSkill, TPromptTemplate, AgentTool>, "model"> & {
+> = Omit<AgentHarnessOptions<TSkill, TPromptTemplate, AgentTool>, "model" | "tools"> & {
/** Kernel browser session used by default CUA tools. */
browser: KernelBrowser;
/** Kernel SDK client used by default CUA tools. */
client: Kernel;
/** Model used by the harness. CUA refs are resolved before pi sees the model. */
model: CuaRuntimeInput;
+ /** Add your own pi tools alongside the built-in browser tools. */
+ extraTools?: AgentTool[];
+ /** Expose a batch tool so the model can run multiple browser actions in one call. */
+ batchTool?: boolean;
+ /** Expose a helper for browser navigation and URL reads. */
+ computerUseExtra?: boolean;
/** Optional payload hook composed after the provider-specific CUA payload hook. */
onPayload?: SimpleStreamOptions["onPayload"];
};
@@ -89,9 +101,10 @@
/**
* Holds the CUA-specific pieces that have to change when a model changes.
*
- * If callers omit `tools` or `systemPrompt`, CUA owns those values and refreshes
- * them from `@onkernel/cua-ai` whenever the model changes. If callers pass
- * their own tools or prompt, the controller preserves those caller-owned values.
+ * CUA owns the computer-use tools and refreshes them from `@onkernel/cua-ai`
+ * whenever the model changes. Caller-owned `extraTools` are appended after
+ * those defaults. If callers pass their own prompt, the controller preserves
+ * that caller-owned prompt.
*/
class CuaRuntimeController {
private runtimeSpec: CuaRuntimeSpec;
@@ -101,7 +114,9 @@
browser: KernelBrowser;
client: Kernel;
model: CuaRuntimeInput;
- tools?: AgentTool[];
+ extraTools?: AgentTool[];
+ batchTool?: boolean;
+ computerUseExtra?: boolean;
systemPrompt?: unknown;
onPayload?: SimpleStreamOptions["onPayload"];
},
@@ -114,7 +129,7 @@
}
get ownsTools(): boolean {
- return this.options.tools === undefined;
+ return true;
}
get ownsSystemPrompt(): boolean {
@@ -130,20 +145,80 @@
}
tools(): AgentTool[] {
- return (
- this.options.tools ??
- createCuaComputerTools({
+ return [
+ ...createCuaComputerTools({
browser: this.options.browser,
client: this.options.client,
toolDefinitions: this.runtimeSpec.toolDefinitions,
- })
- );
+ coordinateSystem: this.runtimeSpec.coordinateSystem,
+ screenshot: this.runtimeSpec.screenshot,
+ batchTool: this.options.batchTool,
+ computerUseExtra: this.options.computerUseExtra,
+ }),
+ ...(this.options.extraTools ?? []),
+ ];
}
onPayloadFor(model: CuaRuntimeInput): SimpleStreamOptions["onPayload"] {
const runtimeSpec = resolveCuaRuntimeSpec(model);
- return composeOnPayload(runtimeSpec.onPayload, this.options.onPayload);
+ return composeOnPayload(
+ composeOnPayload(this.screenshotOnPayload(runtimeSpec), this.providerOnPayload(runtimeSpec)),
+ this.options.onPayload,
+ );
}
+
+ keepToolNames(): string[] {
+ return [
+ ...(this.options.extraTools ?? []).map((tool) => tool.name),
+ ...(this.options.batchTool ? [CUA_BATCH_TOOL_NAME] : []),
+ ...(this.options.computerUseExtra ? [CUA_NAVIGATION_TOOL_NAME] : []),
+ ];
+ }
+
+ private providerOnPayload(runtimeSpec: CuaRuntimeSpec): SimpleStreamOptions["onPayload"] | undefined {
+ if (!runtimeSpec.onPayload) return undefined;
+ return async (payload, model) =>
+ runtimeSpec.onPayload?.(payload, model as Model<Api>, { keepToolNames: this.keepToolNames() });
... diff truncated: showing 800 of 3979 linesYou can send follow-ups to the cloud agent here.
rgarcia
force-pushed
the
hypeship/yutori-native-tools
branch
from
May 20, 2026 15:23
360dba8 to
810ad63
Compare
rgarcia
force-pushed
the
hypeship/yutori-native-tools
branch
4 times, most recently
from
May 20, 2026 16:11
9e69706 to
350ca08
Compare
rgarcia
force-pushed
the
hypeship/yutori-native-tools
branch
from
May 20, 2026 16:38
350ca08 to
53f7e28
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Tzafon hscroll scrolls at origin
- The
hscrollnormalization now preserves optionalx/ycoordinates when building canonicalscrollactions, preventing default translation to(0,0).
- The
Or push these changes by commenting:
@cursor push 75aebfccd6
Preview (75aebfccd6)
diff --git a/packages/ai/src/providers/tzafon/provider.ts b/packages/ai/src/providers/tzafon/provider.ts
--- a/packages/ai/src/providers/tzafon/provider.ts
+++ b/packages/ai/src/providers/tzafon/provider.ts
@@ -207,7 +207,14 @@
case "scroll":
return [toScrollAction(current)];
case "hscroll":
- return [{ type: "scroll", scroll_x: readOptionalNumber(current, "scroll_x") ?? readOptionalNumber(current, "amount") ?? 0 }];
+ return [
+ {
+ type: "scroll",
+ x,
+ y,
+ scroll_x: readOptionalNumber(current, "scroll_x") ?? readOptionalNumber(current, "amount") ?? 0,
+ },
+ ];
case "navigate":
return [{ type: "goto", url: getString(current, "url") }];
case "wait":You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit 53f7e28. Configure here.
| case "scroll": | ||
| return [toScrollAction(current)]; | ||
| case "hscroll": | ||
| return [{ type: "scroll", scroll_x: readOptionalNumber(current, "scroll_x") ?? readOptionalNumber(current, "amount") ?? 0 }]; |
There was a problem hiding this comment.
Tzafon hscroll scrolls at origin
Medium Severity
When Tzafon returns a native hscroll computer action, normalization builds a canonical scroll with only scroll_x and no pointer position. The agent translator then sends the scroll to Kernel at viewport coordinates (0, 0) instead of the intended location, so horizontal scrolling can hit the wrong UI region.
Reviewed by Cursor Bugbot for commit 53f7e28. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tool_setmode instead of sending default CUA batch/navigation browser toolsTests
npm run build --workspace @onkernel/cua-ainpm test --workspace @onkernel/cua-ainpx tsc -bnpm test --workspace @onkernel/cua-agentYUTORI_API_KEY=<set> npx vitest --run test/batch-tool.integration.test.ts -t yutorifrompackages/ainpm run buildwas not completed because the optional@onkernel/ptywrightnative build requireszig; TypeScript compilation passed vianpx tsc -bNote
Medium Risk
Moderate risk because it changes tool schemas/names and provider payload middleware (Yutori/Tzafon) plus adds screenshot transformation/coordinate normalization in the agent executor, which can break existing integrations and runtime behavior across providers.
Overview
Switches CUA from a batch/navigation-centric tool surface to canonical per-action tools (e.g.
click,screenshot) via the newcomputerTools()APIs and updated docs/examples/tests.Updates provider runtime specs and payload adapters:
resolveCuaRuntimeSpec()now returns per-providercoordinateSystem, optionalscreenshotpolicy (Yutori appends a transformed WebP screenshot to the latest message), and a new payload hook context (keepToolNames) so providers can strip local executor tools while preserving caller tools. Yutori now uses nativetool_setmode with native→canonical action normalization; Tzafon now injects its nativecomputer_usetool and mapscomputer_calloutputs into canonical tool calls.Enhances
@onkernel/cua-agentexecution: addsextraTools, optional synthesizedbatch_computer_actions/computer_use_extra, executes individual action tools locally, normalizes key combos, supports provider coordinate denormalization, normalizes baregotoURLs, and adds screenshot transforms (via newsharpdependency). CI integration coverage is updated to run the newcomputer-tool.integration.test.ts.Reviewed by Cursor Bugbot for commit 53f7e28. Bugbot is set up for automated code reviews on this repo. Configure here.